1. Field of the Invention
The present invention relates to a device and method for processing image data, a transmitting medium, and a recording medium. More particularly, the present invention relates to a device and method for processing image data, a transmitting medium, and a recording medium, in which a watermark is blocked, thereby the watermark can be certainly appended to coded image data and can be detected.
2. Description of the Related Art
There is existing a technology of appending some accompanying information associated with the image data into a particular image data (of a static image or a time-varying image sequence) and detecting the accompanying information when reproducing. A representative example thereof is appendage of information regarding the copyright.
In the case where there is a possibility that unspecified users can use particular image data, it is necessary to append information regarding the copyright of the image into the image data in order that a copyright holder of the image asserts his copyright. By appending copyright information in advance, it becomes possible to take measures such as disabling display of the image data when copyright information indicative of inhibiting display of the image data is detected during processing in an image reproducing apparatus or a data reproducing method.
The aforementioned appendage and detection of copyright information is currently much used for preventing the unauthorized copying of videotapes and the like. Recently, there are many video-rental shops lending videotapes. If many of users make an unauthorized duplication of the videotape lent from a video-rental shop at a low price and enjoy the duplicated recordings, economical damages of the copyright holders of the videotape and the video rental shops are significant.
Since image data is recorded on videotape in an analogue fashion, the quality of the copied image is more or less degraded. Accordingly, if such copying is repeated a plurality of times, maintaining the same image quality as the first copy is very difficult.
On the other hand, in a reproducing apparatus and the like which records and reproduces image data in a digital fashion, economical damages produced by the unauthorized copying becomes more serious. This is because that the image quality is not degraded fundamentally by copying in a device or the like that manipulates image data in a digital fashion. Therefore, prevention of the unauthorized duplication in a device or the like that employs digital fashion processing is much more important compared with an apparatus or the like performing the analogue fashion processing.
Mainly, there are two methods for appending accompanying information associated with image data into the image data.
In accordance with the first method, the accompanying information is appended to an auxiliary part of the image data. In videotape, for example, the auxiliary information of the image data is recorded at the top (auxiliary part) of the picture that is substantially invisible when viewed. It is possible to append the accompanying information by using a part of this area.
In accordance with the second method, the accompanying information is appended to a main part (substantially visible part) of the image data. This is done by appending a particular pattern (watermark pattern) to the whole or a part of the image to an invisible extent. An example thereof is spread spectrum communication in which information is appended or detected using a key pattern generated through the use of random numbers, a maximum length sequence, etc.
Hereinafter, a description will be made of one example of appendage of the accompanying information to a main part of image data or detection thereof when a watermark pattern is used. FIG. 26 to FIG. 29 show one embodiment thereof.
Here, it is assumed that a watermark pattern corresponding to 4nxc3x974n pixels as shown in FIG. 26 is used. Each pixel of the watermark pattern takes on a plus or minus symbol. It is desirable that the pixels take on either one of these two symbols at random, and the shape and size of the area may be optional.
For appending a watermark, an area having the same size as the watermark pattern area is selected from the image to be subjected to the appendage. The selected area is superimposed over the watermark pattern and compared therewith. Value xe2x80x9caxe2x80x9d is added to each of pixels corresponding to the plus symbol, and value xe2x80x9cbxe2x80x9d is subtracted from each of pixels corresponding to the minus symbol. Both values xe2x80x9caxe2x80x9d and xe2x80x9cbxe2x80x9d may be optional, but each of the values should be constant through the watermark pattern.
In the example shown in FIG. 27 to FIG. 29, it is assumed that a=1 and b=1. When values of the pixels to be subjected to the appendage are all 100 as shown in FIG. 27, pixel values of 101 and 99 are generated by an embedding operation corresponding to the watermark.
For detecting a watermark, an area having the same size as the watermark pattern is area having the same size as the watermark pattern area is selected from the image that is to be subjected to the appendage. The value obtained by adding all the pixel values in this area is used as an evaluation value. For adding all the pixel values, the selected area is superimposed over and compared with the watermark pattern. Then each of the pixels corresponding to the plus symbol is subjected to addition and each of the pixels corresponding to the minus symbol is subjected to subtraction. In the example shown in FIG. 28, the pixels having a pixel value of 101 are subjected to addition while the pixels having a pixel value of 99 are subjected to subtraction, and all the results thereof are added. At this time, unless the same watermark pattern as that of used for appending the watermark is used, the watermark can not be detected properly. According to the aforementioned detection operation, the evaluation value when the watermark is appended becomes (4n)2 (that is equal to the number of the pixels contained in the area) as shown in FIG. 28, and the evaluation value when the watermark is not appended becomes 0 as shown in FIG. 29.
When the area of the watermark pattern is large enough and the watermark pattern is arranged in a sufficiently random fashion, the evaluation value when no watermark is appended always becomes about 0. Therefore, when the evaluation value exceeds a particular threshold value, it is determined that any watermark is appended. According to the procedure mentioned above, it becomes possible to append binary data (1 bit) indicating whether or not the watermark is appended. When much more information is desired to be appended, the entire image is divided into k areas and each of the areas is subjected to proper processing such as performing the foregoing operations, thereby 2k (k bit) piece of information can be appended.
For example, a watermark pattern generated through M-sequences may be used. The M-sequence (maximum length sequence) is a number sequence consisting of binary symbols of 0 or 1. The statistical distribution of 0""s and 1""s are homogeneous. A correlation code at zero point is 1, and the others are inversely proportional to a code length. It is of course that the watermark pattern may be used through the use of other method than that of using the M-sequence.
When recording and reproducing image data in a digital fashion, the image data is generally compressed since the information volume thereof becomes considerably abundant as it is. High-efficiency coding methods such as JPEG (Joint Photographic Experts Group) (color still-picture coding method), MPEG (Moving Picture experts Group) (color moving-picture coding method) and the like are standardized internationally as the image data compressing methods and put to practical use. Next, a description will be made of a configuration example of appending and detecting the accompanying information, taking the case of compressing image data in accordance with this high-efficiency coding.
FIG. 30 shows a configuration of an encoder. An accompanying information signal f is looked up when given image data is supplied to a watermarking device 1. Whether the processing to append the watermark is performed is determined depending on the obtained accompanying information signal f. Then the image data is supplied to a coder 2 where high-efficient coding is carried out for generating the corresponding code bit string.
FIG. 31 shows a more specific configuration example of the coder 2. In this example, the coder 2 comprises a frame memory 41 that is adapted to storing the image data supplied from the watermarking device 1 frame by frame. A motion vector detector 50 detects a motion vector v from the image data stored in the frame memory, and supplies the results both to a motion compensator and a variable length coding device 46. In the motion vector detector 50, a block matching operation is carried out in macro blocks. Each macro block consists of 16xc3x9716 pixels. Then the motion vector v is detected. In order to increase accuracy, a matching operation in half-pixels is also carried out.
The motion compensation device 43 incorporates therein a frame memory. Using the image which was once coded, then encoded, and stored in the incorporated frame memory, the motion compensation device predicts pixel values at respective positions in the current frame to be coded. A predicted value Ixe2x80x2 {i, j, t} of a pixel value I {i, j, t} at a frame position (i, j) which is supplied at the time xe2x80x9ctxe2x80x9d is determined using the motion vector v=(vx (i, j, t), vy (i, j, t)) by the following equation:
Ixe2x80x2{i, j, t}=(I{ixe2x80x2, jxe2x80x2, txe2x88x92T}+I{ixe2x80x2+1, jxe2x80x2, txe2x88x92T}+I{ixe2x80x2, jxe2x80x2+1, txe2x88x92T}+I{ixe2x80x2, jxe2x80x2+1, txe2x88x92T})/4.
Here, ixe2x80x2 and jxe2x80x2 are given by the following equations:
ixe2x80x2=int(i+vx(i, j, t)T),
jxe2x80x2=int(j+vy(i, j, t)T),
where T is a difference between the time when an image I to be currently predicted is supplied and the time when an image memory already stored in the frame memory is supplied. The right side of the above expression I{ixe2x80x2, jxe2x80x2, txe2x88x92T}+I{ixe2x80x2+1, jxe2x80x2, txe2x88x92T}+I{ixe2x80x2, jxe2x80x2+1, txe2x88x92T}+I{ixe2x80x2, jxe2x80x2+1, txe2x88x92T} represents the pixel values stored in the frame memory which is stored in the motion compensator 43, and int(x) represents the maximum integer not greater than x.
A subtracter 42 subtracts the predicted value obtained by performing motion compensation in accordance with the motion vector v which is supplied from the motion compensator 43 from the image value to be currently coded which is supplied from the frame memory 41. A DCT device 44 subjects an 8xc3x978 pixel block consisting of difference values supplied from the subtracter 42 to two-dimensional DCT (Discrete Cosine Transform). A quantizer 45 uses an appropriate step size Q for quantizing DCT coefficient c supplied from the DCT device 44 following the next equation:
cxe2x80x2=int(c/Q).
The DCT coefficient cxe2x80x2 obtained through the quantization by the quantizer 45 is supplied to both a variable length coding device 46 and an inverse quantizer 47. The variable length coding device 46 subjects both the DCT coefficient cxe2x80x2 which is quantized by the quantizer 45 and the motion vector v supplied from a motion vector detector 40 to variable length coding, and feeds a resultant code bit string.
The inverse quantizer 47 makes inverse quantization processing as is given by the following equation using the step size Q that is same as the step size used by the quantizer 45:
cxe2x80x3=cxe2x80x2xQ
The data obtained by the inverse quantization by the inverse quantizer 47 is supplied to an IDCT device 47 in which the data is subjected to inverse DCT processing and the pixel difference value is restored.
The difference value supplied from the IDCT device 48 is added by an adder 49 to the predicted value supplied from the motion compensator 48 and restored to its original pixel value data. The resultant data is then stored in the frame memory incorporated in the motion compensator 43.
Generally, inter-picture coding (INTER coding) is carried out for coding the difference from the predicted value as mentioned above. However, when the difference between the pixel value to be currently coded and the predicted value calculated by the motion compensator 43 is large, intra-picture coding (INTRA coding) may be carried out for prevention of increase of the code bit amount. That is, each pixel value in the block is supplied to the DCT device 44 without performing a difference calculation, where the pixel value is subjected to coding.
FIG. 32 shows a decoder. A supplied code bit string is restored to the image data in a decoder 21. Then, a watermark detector 22 detects an accompanying information signal f.
FIG. 33 shows a detailed configuration example of the decoder 21. In this configuration example, a variable length decoding device 61 of the decoder 21 subjects the supplied code bit string to variable length decoding processing, then supplies the resultant restored image data (DCT coefficient) to an inverse quantizer 62, and transmits the motion of the decoded motion vector v to a motion compensator 65. The inverse quantizer 62 subjects the supplied DCT coefficient to inverse quantization, and sends the result to IDCT device-63. The IDCT device 63 subjects the inverse-quantized DCT coefficient to IDCT processing, thereby restores the image data having the original differential value, and then supplies the result to an adder 64.
The motion compensator 65 subjects the image data stored in the incorporated frame memory to motion compensation in accordance with the motion vector v supplied from the variable length coding device 61, and generates and supplies a predicted image to the adder 64. Then the adder 64 adds to the predicted image the differential value received from the IDCT device 63, and thereby restores and supplies the original frame image.
The output data from the adder 64 is supplied and stored in a frame memory incorporated in the motion compensator 65, and also supplied to the water mark detector 22. The watermark detector 22 detects and transmits the accompanying information signal f from the received image data and supplies the original image data.
If the image data is not subjected to high-efficiency coding, it is appropriate to use the configurations identical with those of FIG. 30 and FIG. 32 except that the coder 2 and decoder 21 are omitted respectively.
Configurations of a watermarking device 1 and a watermark detector 22 are illustrated in FIG. 34 and FIG. 36, respectively.
FIG. 34 shows a configuration of the watermarking device 1. Supplied image data and an accompanying information signal f to be referred to are both transmitted to a watermark image generator 11. If the accompany information signal f is on, then a watermark is appended to each of pixels in a target image area. For this purpose, first, the watermark image generator 11 transmits to a matching controller 101 a position p of a target pixel to be appended. For example, this pixel position p may be represented in one-dimensional expression by indicating the ordinal number of pixel position when counting from an upper left position of the image in a scanning order. The watermark pattern matching controller 101 looks up a symbol of the watermark pattern stored in a watermark pattern holding memory 12 based on the pixel position p, and passes the obtained symbol S to the watermark image generator 11. The watermark image generator 11 uses the supplied symbol S to append the watermark to a target pixel.
FIG. 36 shows a series of processing carried out in the watermarking device 1. First, in step S141, watermarking levels a and b are set to predetermined values. In step S142, an area having the same size as the watermark pattern area is selected from a target image, and each pixel in the selected area is matched against the watermark pattern. In step S143, a symbol of the watermark is discriminated, and if the symbol of the watermark pattern corresponding to the pixel is plus, then the value a is added to the pixel in step S144. On the other hand, if the symbol of the water pattern corresponding to the pixel is minus, then the value b is subtracted from the pixel in step S145. The foregoing processing is repeated until step S146 determines that all the pixels in the target area have been processed.
FIG. 35 shows a configuration of the watermark detector 22. Supplied image data is given to an estimation value calculator 31. In order to obtain an estimation value from each pixel on the target image area, the estimation value calculator 31 transmits an image position p to be estimated to the watermark pattern-matching controller 111. As mentioned above, this image position p is represented in one-dimensional expression by indicating the ordinal number of pixel position when counting from an upper left position of the image in a scanning order. The watermark pattern matching controller 111 looks up a symbol of the watermark pattern stored in the watermark pattern holding memory 32 based on the pixel position p, and passes the obtained symbol S to the watermark image generator 31. The watermark image generator 31 uses the supplied symbol S to calculate the estimation value. The resultant estimation value is subjected to thresholding in an estimation value comparator 33, and the accompanying information signal f is supplied. Besides, the supplied image data is outputted through an image converter 34 as it is or after subjected to predetermined handling or processing.
FIG. 37 shows a series of processing carried out in the watermark detector 22. First, in step S161, an estimation value sum is initialized and a threshold value xe2x80x9cthxe2x80x9d is set. In step S162, an area equal to a watermark pattern area in size is selected, and then each element in the selected area is matched against the watermark pattern. If step S163 determines that the symbol of the watermark corresponding to the pixel is plus, then the pixel value is added to the evaluation value sum in step S164. On the other hand, if the symbol of the water pattern corresponding to the pixel is minus, then the pixel value is subtracted from the evaluation value sum in step S165. The foregoing processing is repeated until step S166 determines that all the pixels in the target area have been processed. After that, the evaluation value sum is compared with the threshold value xe2x80x9cthxe2x80x9d in step S167. If the result is sum greater than th, then it is determined that the watermark is appended, and an accompanying information signal f is turned on in step S168. If not, then the accompanying information signal f is turned off in step S169.
The accompanying information signal f is used for prevention of the unauthorized copying as follows. Taking the decoder in FIG. 32 as an example, image data and an accompanying information signal f to be transmitted are passed to an image display section not shown. The image display section displays the image as it is if the accompanying information signal f is on. However, if the accompanying information signal f is off, then the display section carries out various handling or processing such as not displaying images, not displaying main area of the image data, scrambling the image (displaying received image data at random), and so on. Alternatively, an image converter 34 disposed in the watermark detector 22 as shown in FIG. 35 may be provided for carrying out the foregoing image data handling or processing depending on the accompanying information signal f.
The aforementioned two methods for appending information associated with image data into the image data come with problems mentioned below.
In the first method for appending an accompanying information to an auxiliary part of image data, if the auxiliary part to which the accompanying information is appended is ignored, then it becomes different to prevent unauthorized copying. For example, if one allows a commercially available personal computer to read therein image data that is recorded in a digital fashion, ignores the auxiliary part of the image and extracts only the main part thereof, the image quality becomes precisely identical with that before copying. In this case, appending the accompanying information to the auxiliary part becomes useless.
In the second method, appended accompanying information does not become lost, for example, by the copying procedure as mentioned in connection with the foregoing first method. However, when image data is subjected to various signal processing such as noise reduction filtering or the like, the components of the appended accompanying information can be attenuated and become impossible to be extracted.
Particularly, in the case that original image data itself is compressed in accordance with high-efficiency coding such as JPEG, MPEG, or the like, an adverse affect caused by quantization processing may be exerted. Owing to quantization of the high-efficiency coding, components of the appended accompanying information are amplified to an extent to be visible, thereby the image quality becomes degraded, or attenuated to the extent that they can not be extracted, and its original meaning can be lost.
There is another method to add the accompanying information using a special area on the image so that the components of the accompanying information are not changed by the aforementioned signal processing. From the viewpoint of a whole image sequence, however, such area is provided at only one part thereof, and therefore it is impossible to reserve a sufficient large area for a watermark pattern. For this reason, the evaluation value takes on a large value other than zero even when no accompanying information is added. As a result, when an absolute evaluation criteria is used to judge that some accompanying information is appended if a predetermined threshold value is exceeded, detection of the accompanying information becomes very difficult.
In addition, when the area to which the accompanying information is appended is only one part from a viewpoint of whole image sequence, it becomes considerably difficult to append plural pieces of information. For example, when the whole image is divided into k areas and then the accompanying information is appended to each of these areas, a watermark pattern area for each of these areas becomes smaller depending on the number of these areas, and therefore detection of the accompanying information becomes substantially impossible.
These problems mentioned above are particularly remarkable in time-varying image sequence.
With the above problems in view, an object of the present invention is to enable accompanying information to be detected certainly without exerting a significant influence to original image data.
A device and method for processing image data, and a transmitting medium of the present invention described here are characterized in that, in image data processing by which accompanying information is embedded as a watermark into coded image data, a position of the coded image data in a block as a coding unit is detected, a blocked watermark pattern is provided, an area in which an operation relative to a first level value is performed and an area in which an operation relative to a second level value is performed are offered, and a watermark is appended to the coded image data in accordance with the blocked watermark pattern.
Further, a device and method for processing image data, and a transmitting medium of the present invention are characterized in that, in image data processing by which image data having accompanying information embedded as a watermark is processed, a position of the coded image data in a block as a coding unit is detected, a blocked watermark pattern is generated, an area in which an operation relative to a first level value is performed and an area in which an operation relative to a second level value is performed are offered, an evaluation value of the coded image is calculated in accordance with the blocked watermark pattern, and the evaluation value is compared with a predetermined reference value.